Integration of Speech and Gesture Inputs during Multimodal Interaction

Speech and gesture are two types of multimodal inputs that can be used to facilitate more natural humanmachine interaction in applications for which the traditional keyboard and mouse input mechanisms are inappropriate, however the possibility of their concurrent use raises the issue of how best to fuse the two inputs. This paper analyses data collected from a speech and manual gesture-based digital photo management application scenario, and from this derives assumptions and fusion thresholds with which future speech/gesture systems can be designed. Gesture input was found to overlap with speech input in nearly all multimodal constructions (95%), and was completely subsumed by speech input in most multimodal constructions (56%), in distinct contrast to previous similar analyses for combined pen and speech input, in which the pen input frequently precedes the speech input.